5 . Recurrent networks

So far we have been studying networks that only have forward connections and thus there have been no loops. Networks with loops — recurrent connections is the technical term used in the literature — have an internal dynamics. The networks that we have introduced so far have only retained information about the past in terms of their weights, which have been changed according to the learning rules. Recurrent networks also retain information about the past in terms of activation levels: activation is preserved for a certain amount of time: it is " passed back " through the recurrent connections. This leads to highly interesting behaviors, behaviors that can be characterized in terms of dynamical systems. We will therefore briefly introduce some elementary concepts from the area of dynamical systems. The most important type of network that we will discuss in this chapter are the Hopfield nets. Because of their properties, they have inspired psychologist, biologists, and physicists alike. Associative memories — also called content-addressable memories — can be realized with Hopfield nets, but also models of physical systems like spin-glasses can be modeled using Hopfield networks. Content-addressable memories are capable of reconstructing patterns even if only a small portion of the entire pattern is available. We will first discuss Hopfield nets with all their implications. We will end with a short note on other kinds of recurrent networks. Associative memory is one of the very fundamental problems of the field of neural networks. The basic problem of associative memory can be formulated as follows: " Store a set of p patterns ξ µ in such a way that when presented with a new pattern ζ , the network responds by producing whichever one of the stored patterns most closely resembles ζ. " (Hertz et al., 1991, p. 11). The set of patterns is given by {ξ 1 , ξ 2 ,…,ξ p }, the nodes in the network are labeled 1, 2, …, N. A pattern of activation in Hopfield nets always includes all the nodes. The patterns are binary, consisting of values {0,1} or alternatively {-1,+1}. The former can be translated into the latter as follows. Let n i be values from the set {0,1}. These values can be transformed into {-1,+1} simply by S i = 2n i − 1. The symbol S i always designates units that assume values {-1,+1}. Remember, how a Hopfield net is constructed? …